Schemas v0.1 v0.2 #76

will-moore · 2021-12-01T13:48:03Z

This adds schemas, tests and samples for JSON schema support of v0.1 and v0.2.

Updates all schemas to NOT require version, name etc, since these are only SHOULD in the spec (noticed this when validating some existing v0.1 files).

Also adds plate samples and plate.schema for the HCS spec (tests for 0.1 only, since this hasn't changed since).

This also adds a way of specifying the SHOULD rules within the schema, using a second schema with allOf to add restrictions to the image.schema (see below - thanks @sbesson).
This more strict schema can be used to validate without failing on the first Error, allowing it to be used for reporting multiple suggestions for rules that SHOULD be obeyed, but aren't actually invalid.

Added tests that check that all valid files (except image_complete.json) give at least one SHOULD warning.

{
  "allOf": [
    {
      "$ref": "http://localhost:8000/image.schema"
    },
    {
      "properties": {
        "multiscales": {
          "items": {
            "required": [
              "version", "metadata", "type"
            ]
          }
        }
      }
    }
  ]
}

will-moore · 2021-12-03T10:28:02Z

That last commit avoids the need to run httpserver in the test, which will be nice for ome-zarr-py validate use-case where we want to ship the schemas with the library and not load them from a URL each time (or have to serve them from httpserver).
However, it does require that the code has some knowledge of the schema to update appropriate $ref:

schema['allOf'][0]['$ref'] = IMAGE_SCHEMA_KEY

will-moore · 2021-12-03T11:17:04Z

The last commit uses a better approach of subclassing the RefResolver to always load local schemas instead of remote.

joshmoore

In general, love all the examples and the fact that this covers all current versions. One general naming consideration about the URLs since those are something we want to last. I haven't (yet) gone through each example and schema, but I don't think I'll get to that before Thursday. If it helps to get this in here so that ome-zarr-py etc. can build on top of it, no objections. We should just review before we make the URLs live. (I assume at this point they don't actually get built into the spec page)

joshmoore · 2021-12-01T14:09:09Z

0.1/examples/image/invalid/missing_version.json

@@ -2,7 +2,6 @@
    "@type": "ngff:Image",
    "multiscales": [
        {
-            "version": "0.2",


This was dropped from v0.3?

Fixed to "0.1"

joshmoore · 2021-12-06T13:32:08Z

0.3/schemas/json_schema/image.schema

@@ -1,6 +1,6 @@
 {
  "$schema": "https://json-schema.org/draft/2020-12/schema",
-  "$id": "http://localhost:8000/image.schema",
+  "$id": "https://ngff.openmicroscopy.org/0.3/schemas/json_schema/image.schema",


The "schema(s)" in this URL are a bit much....

That is just the path we currently have in this repo. Since we're only supporting json_schema now, we could drop that dir. But the paths are even more painful at ome/ome-zarr-py#142 where I wanted to maintain the same directory structure as here, simply replacing https://ngff.openmicroscopy.org/ with a single path in that repo (expecting that we'll package the schemas as part of the release). So the path in that repo then becomes schemas/0.3/schemas/json_schema/image.schema!
The problem with is that in the NGFF repo, the version is at the top level, so we have a schemas directory under each version, but it's the other way around on ome-zarr-py where I wanted all the schemas to be under a top-level schemas/.
Need to unify the approach between the repos. Do we create top-level version dirs on ome-zarr-py?

will-moore · 2021-12-09T12:22:36Z

Once we've "published" a schema to a public URL that people can use for validation, can we make any future changes to that schema, and how do we version it?
E.g. if we make a change/fix and then an instance file becomes valid/invalid as a result??

will-moore · 2021-12-10T12:18:12Z

From the discussion this morning a plan of action:

Move schemas to e.g. /0.3/schemas/image.schema (remove extra json_schema dir)
Make ngff pip installable
Include get_schema(version, strict=False) from ome-zarr-py validation PR
Update bikeshed job to also publish schemas - see https://github.com/joshmoore/spec-prod

I think only the first point should be done in this PR?

joshmoore · 2021-12-13T16:02:04Z

Merging as discussed this morning. As mentioned, we can re-review the URLs before going public with them.

@joshmoore

Schemas v0.1 v0.2 SHA: a20dd71 Reason: push, by @joshmoore Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

will-moore added 9 commits December 1, 2021 10:20

Remove v0.2 support from 0.3/schemas/json_schema/image.schema

2018d3a

Add validation tests and schemas for 0.1 and 0.2

2eaf0c8

Add plate schema and examples for v0.1

880c650

Test for SHOULD rules

e9b5560

multiscales 'name' is not required

df8962e

version is not required property 0.1, 0.2, 0.3

20481ca

v0.1. images SHOULD contain 'type' but not 'omero'

145b0ad

Remove unused 'multiscales' from strict_image.schema

f392500

Remove non-essential 'version' and 'name' from v0.3 image.json

8449821

will-moore mentioned this pull request Dec 2, 2021

Initial validate support, with --strict option ome/ome-zarr-py#142

Closed

will-moore added 3 commits December 2, 2021 16:40

Use allOf in strict_image.schema to add SHOULD rules to image.schema

4f2bf33

Remove unused merge()

c3283d1

Manually cache image.schema instead of load via URL resolver

c2c1394

Subclass RefResolver to load local files

7513ddc

will-moore added 2 commits December 3, 2021 16:30

Add 'name' to strict_image.schema. multiscales SHOULD have name

37c56dd

Improve LocalRefResolver - overwrite resolve_remove()

e4865a6

joshmoore reviewed Dec 6, 2021

View reviewed changes

Fix omero.version numbers in 0.1 samples

78c658f

will-moore added 2 commits December 10, 2021 14:14

remove json_schema/ directory from each version

7ec782c

Remove .DS_Store

4aeb82b

joshmoore merged commit a20dd71 into ome:main Dec 13, 2021

sbesson added this to the 0.4 milestone Feb 1, 2022

sbesson mentioned this pull request Mar 31, 2022

NGFF dataset validator #58

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Schemas v0.1 v0.2 #76

Schemas v0.1 v0.2 #76

will-moore commented Dec 1, 2021 •

edited

Loading

will-moore commented Dec 3, 2021

will-moore commented Dec 3, 2021

joshmoore left a comment

joshmoore Dec 1, 2021

will-moore Dec 9, 2021

joshmoore Dec 6, 2021

will-moore Dec 9, 2021

will-moore commented Dec 9, 2021

will-moore commented Dec 10, 2021

joshmoore commented Dec 13, 2021

Schemas v0.1 v0.2 #76

Schemas v0.1 v0.2 #76

Conversation

will-moore commented Dec 1, 2021 • edited Loading

will-moore commented Dec 3, 2021

will-moore commented Dec 3, 2021

joshmoore left a comment

Choose a reason for hiding this comment

joshmoore Dec 1, 2021

Choose a reason for hiding this comment

will-moore Dec 9, 2021

Choose a reason for hiding this comment

joshmoore Dec 6, 2021

Choose a reason for hiding this comment

will-moore Dec 9, 2021

Choose a reason for hiding this comment

will-moore commented Dec 9, 2021

will-moore commented Dec 10, 2021

joshmoore commented Dec 13, 2021

will-moore commented Dec 1, 2021 •

edited

Loading